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Abstract 

Using probabilistic approach, the transient dynamics of sparsely connected Hopfield 
neural networks is studied for arbitrary degree distributions. A recursive scheme is 
developed to determine the time evolution of overlap parameters. As illustrative 
examples, the explicit calculations of dynamics for networks with binomial, power- 
law, and uniform degree distribution are performed. The results are good agreement 
with the extensive numerical simulations. It indicates that with the same average 
degree, there is a gradual improvement of network performance with increasing 
sharpness of its degree distribution, and the most efficient degree distribution for 
global storage of patterns is the delta function. 
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As a tractable toy model of associative memories and can also be viewed as 
an extension of the Ising model, Hopfield neural networks pQ received lots 
of attention in recent two decades. Equilibrium properties of fully-connected 
Hopfield neural network have been well studied using spin-glass theory, es- 
pecially the replica method [2|3] . Dynamics is also studied using generating 
functional method jl] and signal-to-noise analysis [5|6|7] . 

Given the huge number of neurons, there is only small number of intercon- 
nections in human brain cortex (~ 10 11 neurons and ~ 10 14 synapses). In 
order to simulate a biological genuine model rather than the fully-connected 
networks, various random diluted models were studied, including extremely 
diluted model [HE] , finite diluted model [TOlfTT] . and finite connection model 
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[T2"][T3] . But neural connectivity is suggested to be far more complex than fully 
random graph, e.g. the networks of c.elegans and cat's cortical neural were re- 
ported to be small- world and scale-free, respectively [TlfTS] . To go one step 
closer to more biological realistic model, many numerical studies are carried 
out, focusing on how the topology, the degree distribution, and clustering co- 
efficient of a network topology affect the computational performance of the 
Hopfield model [T6lll7lll8|rT9] . With the same average connection, random net- 
work was reported to be more efficient for storage and retrieval of patterns 
than either small- world network or regular network [TTj . Torres et al. reported 
that the capacity of storage is higher for neural network with scale-free topol- 
ogy than for highly random diluted Hopfield networks [18]. However, to our 
best knowledge, there are no any theoretical results of either dynamics or 
statics yet. 

The goal of this paper is to analytically study the dynamics of Hopfield model 
for a sparsely connected topology whose degree distribution is not restricted 
to a specific distribution (e.g. Poisson) but can take arbitrary forms. Another 
question investigated in this paper is how the degree distribution of connection 
topology influences the network performance, especially whether there exists 
an optimal degree distribution given a fixed number of nodes and connections. 

Let us consider a system of iV spins or neurons, the state of the spins takes 
Si (t) = ±1 and updates synchronously with the following probability, 

J3si(t+l)hi(t) 

p ro b M< + 1 )|M<)H 2cosh(/3Mi)) . ID 



where (3 is the inverse temperature and the local field of neuron i is defined 
by 

N 

hi (t) = £ Vi (*) • (2) 

i=i 



We store q = aN random patterns = (£f , . . . , in networks, where a is 
called the loading ratio. The couplings are given by the Hebb rule, 



Ja = J £ (3) 



where CV,- is the adjacency matrix (Cy = 1 if j is connected to i, Cy = 
otherwise). In contrast to spin glasses or many other physical systems, the 
interactions between biological neurons are not symmetric: neuron i may in- 
fluence neuron j even if neuron j has no influence on neuron i. So in our model, 
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Cij and Cji are chosen independently. Degree of spin i, ki = J2j=i Cij, denotes 
the number of spins that are connected to i. We consider the case that neurons 
are sparsely connected, it means that N — > oo, ki — > oo but ki/N — * 0. For 
example, we can take ki = O(lniV). And in this paper, the degrees of neurons 
are set as an arbitrary distribution p (ki = k). 

We use g (•) to express the transfer function, 

Sl (t + l)=g(hi(t)). (4) 



Without loss of generality, let us consider the case to retrieve We define 
m (t) as the overlap parameter between network state s (t) and the first pattern 
£ x as 

1 N 

™(t) = -j:$Si(t). (5) 

iV i=i 



Then the local field at time t can be represented by 
i n i q n 

h i W = jyE ^e)^ w + -^EE ^ee^i w , (6) 



where the first term is the signal from ^ and the second one is crosstalk noise 
from other patterns. Our aim is to determine the form of the local field in 
the thermodynamic limit N — > oo. We apply the law of large numbers to 
the signal term and find that it converges to ^Ij^m (t) in the thermodynamic 
limit. To show this point intuitively, we can simply replace the signal term by 
its average, 

JP ^ E^tf^i (t) =& (Ctffa (t)) . (7) 



This formula is exact in the thermodynamic limit because the whole system is 
assumed to be self-averaging. Since and £ x are independent of each other, 
we can write the average of product as the product of average, 

V)) = & Pa) (fa®). (8) 



Using definition of ki together with Eq. (151) , we have (Cij) = ki/N and m (t) = 
£jSj (t)\. So we have following formula, 

^ jfT,C^s s {t)=^m(t). (9) 
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Taking a closer look at the second term in Eq. ([6]), if all the terms in the 
sum (with regard to /i) are independent, we are able to apply the central 
limit theorem to it. As pointed out in Ref. [5], two conditions are essential for 
the independence of terms in the sum: first is that almost all feedback loops 
are eliminated, and the second is that with probability 1, any two neurons 
have different clusters of ancestors, i.e. they will remain independent because 
they receive inputs from two trees which have no neurons in common. In our 
model, because of the sparsely connected architecture together with the high 
asymmetry of synaptic connections, two conditions are both satisfied. Thus the 
second term in Eq. §6§ converges to a zero-mean Gaussian form J\f ^0, ^ q ~^] ki ) 

where ^ 9 ^ ki is the variance of Gaussian noise. Then the local field of neuron 
i can be expressed by 

hdt)=^ni{t) + N(v}^^. (10) 



Note that similar treatment of local field can also be found in [7] . 
Then the average state of neuron i was given formally by 

(*(* + !)> = / dz^y^exp (-z 2 /2) (g ^m(t) + ^ ~ ^ \ j ^ ,(11) 



where stands for averaging over distribution of £,}, and = [6(£ + 1) + 5(£ — 1)] /2. 
When self-averaging is assumed, the average of neuron state in the next time 
can be obtained by taking average over all A" neurons, 

(s (t + 1)) = i £ / dz (2n)-^ exp (g (t) + ) .(12) 

Using the concept of degree distribution, we only need to take average over 
the degree distributions as 



(s(t + l)) = Jdkp(k)Jdz(2n)- 1/2 ex V (-z 2 /2) (g (t) + ^ - ' zjj .(13) 



The overlap parameters are obtained in the similar way, 



m(t + l) = J dkp (k) J dz (2vr)- 1/2 exp (-z 2 /2) (?g ^ A m ( t ) + ^ W 
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When focusing on the most interested case of zero temperature {{3 — > oo) 
transfer function g (•) is replaced by sgn (•). From Eq. (I14p one gets 



m(t+l)= / dkp (k) 



:=-f~oo 

J dz{2<K)- 1/2 exp(-z 2 /2) 



J dz{27i)- 1/2 exp(-z 2 /2) 



+ 



dz {2tc)- 1/2 exp (~z 2 /2) 



dz (27iy 1/2 exp(-z 2 /2 



jm(t)+ 



V(g-l)fc 



z=0 



Then, the last equation can be further simplified to 

m (t) 



m (t + 1) = J dkp (k) erf 



(15) 



(16) 



where 



u 

erf (u) = \j2/n J exp {— x 2 /2^j dx. 



(17) 



This finishes the Signal-to-Noise derivation of overlap parameter at zero tem- 
perature. As long as the degree distribution of network is determined, using 
Eq. (11411171) , one can calculate temporal evolution of overlap parameters up to 
an arbitrary time step. 

Using auxiliary thermal fields j(t) to express the stochastic dynamics [7], it 
is easy to extend the method to arbitrary temperatures by averaging the zero 
temperature results over the auxiliary fields. 



s(t + l) = g(h(t)+"f(t) /P), 



is) 



and the probability density of 7 (t) is given by 
p( 7 (t)) = i(l-tanh 2 ( T (t))). 



(19) 



5 



Q. 



0.0 



1.0- 


x 




X 


X 


1 






X 


X 


0.8- 


X' 








X .-• 


X 




X 






0.6- 


x , 


X 














x ■ 


X 




0.4- 


X . 








X 


- 




0.2- 

















- 1 - 





in theory 
* in simulation 



4 6 

Time 



- r- 

8 



10 



Fig. 1. Time evolution of overlap parameters for Hopfield network with delta func- 
tion degree distribution. Initial overlaps range from 1.0 to 0.1 (top to bottom). N 
is 50000. Each neuron has 100 degrees and 20 patterns are stored in networks. 

For illustrative examples, we apply our theory to networks with some specific 
degree distributions and numerical simulations are performed to verify the 
theoretical results. In all of our numerical simulations, we set N = 5 x 10 4 
and the average degree k = 100, varying only the arrangement of connections. 
Each neuron is connected on average to 0.2% of the other neurons compared 
to ~ 0.1% in the mouse cortex [20J. 

The first numerical experiment is the delta function 

p(k) = 6(k-k), (20) 



which means that every neuron has exactly k connections. In practice, the con- 
nection topology is generated by randomizing a regular lattice which average 
degree is k. Time evolutions of overlap parameters from theory and numerical 
simulations are plotted in Fig. [TJ 

The second degree distribution is binomial distribution which comes from a 
Erdos-Renyi random graph [2TJ (see the left panel of Fig. [2]) 

^^©'HP (2i) 

The temporal evolution of overlap parameters are presented in the right panel 
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Fig. 2. Left panel: the normalized binomial degree distribution of networks. Right 
panel: the temporal evolution of overlap parameters for Hopfield network with de- 
gree distribution shown in the left panel (Erdos-Renyi random graph). Initial over- 
laps range from 1.0 to 0.1 (top to bottom). N = 50000, k = 100, and 20 patterns 
are stored in networks. 




Fig. 3. Left panel: the normalized power-law degree distribution (log-log scale). 
Right panel: the time evolution of overlap parameters for Hopfield network with 
degree distribution plotted in the left panel. Initial overlaps range from 1.0 to 0.1 
(top to bottom). N = 50000, k = 100, and 20 patterns are stored in networks. 

of Fig. [2 

The third one is power-law distribution (see the left panel of Fig. [3]) 

p{k) = \k 2 k~ z , (22) 
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Time 

Fig. 4. Theoretical comparison of time evolutions of overlap parameters in networks 
with the same average degree but different degree distributions. Degree distribution 
of A is the delta function, B is binomial, and C is power-law. The inset shows 
in detail that performance of network with delta function degree distribution is 
slightly better than that with binomial distribution (Erdos-Renyi random graph). 
N = 50000, k = 100, and 20 patterns are stored in networks. 

which is of great importance because it may comes from preferential attach- 
ment in the growth process of neurons [22]. The right panel of Fig. [3] shows 
the temporal evolutions of overlap parameters. 

It is obvious that the theoretical results from our scheme are consistent with 
the simulations for the above degree distributions. Emerging naturally from 
the above statements, which form of degree distribution is the best one? 

To investigate how degree distributions influence the performance of networks, 
we theoretically compare time evolutions of overlap parameters with delta 
function (A), binomial (£>), and power-law degree distribution (C) in Fig. HI 
The inset shows the details near stationary states. It indicates that network 
with delta degree distribution performs slightly better than that with binomial 
distribution. The most rapid degradation in overlap occurs in network with 
power-law distribution. This behavior can be interpreted as follows. Using 
Eq. ([TBI) , it is easy to find that an individual neuron with fewer degrees suffers 
more perturbations from crosstalk noise. In the case of power-law degree dis- 
tribution, degrees are not uniformly distributed in networks and there are too 
many neurons with small number of connections, which leads to negative per- 
formance of the entire networks. However, note that despite the disadvantages 
of power-law distribution, hubs (subset of networks which has higher degrees) 
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Fig. 5. Theoretical comparison of time evolutions of overlap parameters for Hopfield 
networks for uniform degree distributions with different width. Degree distribution 
of A is delta function. Left panel: the uniform degree distributions with various 
width, 50 (B), 100 (C), 150 (D), and 200 (E). The inset in right panel shows in 
detail that the performance of A is slightly better than that of B. N = 50000, 
k = 100, and 55 patterns are stored in networks. 



in networks may be useful for partial storage [17] . 

In addition, for verifying the above statements, we construct the special cases 
that the degree distributions of networks is uniform with various widthes, 50 
(B), 100 (C), 150 (D), 200 (E), and the delta function A which width is 
(see the left panel of Fig. [5]). The temporal evolutions of overlaps are plotted 
in the right panel of Fig. [5j and the inset shows the detailed information 
near stationary states. It was found that the much more widespread degree 
distribution tends to induce worse performance of networks. 

In summary, the transient dynamics of sparsely connected Hopfield model with 
arbitrary degree distributions is studied in this paper. It was found that the 
delta function degree distribution is optimal in terms of network performance, 
and there is a gradual improvement for network performance with increasing 
sharpness of its degree distribution. We would like to emphasize that the model 
investigated in this paper is a simple relaxation to real network topology, 
by neglecting loops in it. But Ref. [23] suggested that the feedback loops 
together with their correlations exist in networks and play important role in 
network dynamics even in the case of sparsely connected systems. It would 
be interesting to investigate Hopfield model with real complicated topology 
influenced both by degree distributions and loops (feedbacks and correlations). 
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